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Abstract: Two small studies, one an eye-tracking study and the other a remote observation study, have 
been conducted to investigate ways to identify two kinds of online learner interactions: users flicking 
through the web pages in “browsing” action, and users engaging with the content of a page in 
“learning” action. The video data from four participants of the two small studies using the OpenLearn 
open educational resource materials offers some evidence for differentiating between ‘browsing’ and 
‘learning’. Further analysis of the data has considered possible ways of identifying similar browsing 
and learning actions based on automatic user logs. This research provides a specification for 
researching the pedagogical value of capturing and transforming logs of user interactions into external 
forms of representations. The paper examines the feasibility and challenge of capturing learner 
interactions giving examples of external representations such as sequence flow charts, timelines, and 
table of logs. The objective users information these represent offer potential for understanding user 
interactions both to aid design and improve feedback means that they should be given greater 
consideration alongside other more subjective ways to research user experience. 

Keywords: Eye-tracking study, think aloud protocol, remote observation, usability logs, 
representations of pedagogy 

Interactive Demonstration: Readers are encouraged to go through some of the learning units 
available at OpenLearn . The website may require some computer applications (e.g. pdf viewer, video 
player, Flash player, etc.) in order to go through some of the materials within OpenLearn such as 
documents, files, videos, audio, etc. 

1 Introduction 

OpenLearn provides an open educational resource that can be used by the user in any way they wish. 
Such flexibility in accessing learning materials in raises questions as to how different users (differing 
in age, in location, in qualifications, etc.) with different purposes (information-seeking, studying a 
course, etc.) interact with online resources. It is a challenge to be able to differentiate different user 
actions from their interactions, particularly, to be able to determine whether users are ‘learning’ from 
their interactions with the site. 

Learner-computer interaction may be indicated by variety of sources: logs, interviews, observations, 
recorded screen activities, etc. (e.g. San Diego & Aczel, 2007; Sheard, Ceddia, & Hurst, 2003). For our 
purposes we need to select appropriate techniques that can be used in researching online computer 
interactions and apply them in laboratory based observation studies, remote observation, and indirect 
observation based on user logs. The structure we are following is an experimental approach to bring 
out methodological issues and provide further specifications based on experience. 

An initial desk study of tools for capture of interaction data with web sites and user trials to assess 
interaction models focused on using eye-tracking and other observation as appropriate. This research 
considered digital technologies such as a ‘non-intrusive’ eye-tracking device, remote desktop sharing 
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tool, screen capture software, digital video cameras, webcams, etc., to find out whether integrated 
analysis of the data from these technologies can provide information about ‘learning’ interactions in 
the context of OpenLeam. 

As a further stage we look at how we can go from the information from our studies to consider how we 
can use the data we have about all of our users, rather than just the observed user. The observations 
based on the eye-tracking study and the observations based on the remote observation study are 
therefore examined to come up with specifications for indirect study of learner interactions based on 
examining logged actions in the OpenLeam systems to classify user actions. Different kinds of 
visualisations using recorded logs (e.g. site visited, time of visits, and so on) are illustrated; 
visualisations which may help interpret differentiate models for learning interaction within OpenLeam. 

2 Researching online learner interactions 

Researching human-computer-interactions sometimes takes place in a laboratory setting as the studies 
require a controlled procedure and set-up; and this set-up can be difficult to prepare and do in natural 
settings (Hall, 2000). Researchers may also find capturing and analysing computer actions challenging 
when using conventional ‘note-taking’ observation techniques (Foster, 1996, Pirie, 1996). An 
alternative technique is the use of video, which can allow capture of simultaneous actions using 
multiple videos, repeated replay, playing at different speeds, post-analysis, multiple observers, etc. 
(Powell, Francisco, & Maher, 2003, Roschelle, 2000). This technique coupled with think aloud 
technique (Ericsson and Simon, 1984) makes a powerful research method. 

Researchers also have an opportunity to explore the opportunities that ‘new’ technologies offer (e.g. 
screen capture, digital video cameras, eye-tracking) to capturing computer-learning interactions. The 
technique amalgamating eye-tracking and think-aloud protocol has provided methodological 
opportunities for describing learning interactions (e.g. San Diego & Aczel, 2007, Hansen, Hauland, & 
Andersen2001). For example, it is possible to describe users interaction while reading through a body 
of texts, represented by ‘eye-marks’ embedded on those texts, generated by the eye-tracking device 
(San Diego & Aczel, 2007). While it is possible to bring users in a lab for this kind of study, this can 
be problematic to do when researching users located at a distance. 

Researching users in their natural setting is less intrusive than studying them in a laboratory 
(Hammersley & Atkinson, 1995). There are other ‘new’ technologies that can help in capturing users’ 
learning-interactions at a distance. Thompson (2003), in a study conducted to test the usability of a 
library website with the aim to improve its design, has attempted to use technologies such as screen 
capture software, online conferencing software, and remote computer desktop-sharing application 
software to capture and record remote computer-interactions. Hosein, Aczel, Clow and Richardson 
(2007) conducted a similar study but in the context of looking at learning with mathematical software. 
These previous studies have suggested that remotely capturing users’ interaction can reduce 
methodological problems associated with the setting and researcher’s presence in the same location. 

Another approach to capturing learner-interactions is by indirect observation. For example, in a 
Learning Activity Management System (Dalziel, 2003), a user interaction with an online activity is 
captured and presented in a ‘learner monitoring environment’. This monitoring environment pictorially 
represents different activities as ‘blocks’ and the colours of the blocks represent users’ log of their 
completion of an activity. This kind of feedback has been found to be useful for teachers as it gives 
teachers the feedback of users’ current state in the activity sequence (Cameron, 2007). However, it 
fails to represent the time users has spent within each activity. Teachers can also benefit from 
modelling interaction in terms of the time (Laurillard, 2006) as users face constraints of completing 
learning activities. Sheard, Cedia and Hurst (2003) have offered an alternative way of researching 
online learning interactions. They have attempted to provide educators some information about online 
learning interactions in order to inform the development and improvement or online course materials. 
They captured the users’ frequency of use of online resources designed to support learning, and the 
time users spend with these resources. However the study did not attempt to transform these logs into 
different representational forms to help inform teachers and researchers consider different kinds of 
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learning interactions based on these logs. The technique of capturing and recording of user interaction 
based on logs potentially offers a less obtrusive technique than laboratory-based and remote 
observation techniques. 

A novel consideration is to look at the possibility to determine different kinds of users’ actions from 
matching observation based on eye-tracking experiences together with the observations that can be 
done remotely; and the extent with which the pattern of actions can also be represented by userlogs 
(Ivory and Hearst, 2001). Previous research has informed us some of the methodological advantage 
and pedagogical value of an approach to capturing user interaction based on videos and logs. However, 
it is a challenge to explore the extent that it is possible to distinguish the difference between users 
flicking through the pages in “browsing” action, and when they are more engaged in “learning” action 
based on these logs. Being able to meet this challenge can help i) reach the majority of online users 
who may not be available for user studies ii) provide a relatively ‘non-obtrusive’ data capture iii) 
inform course design and linked resources. 

The study that is carried out involves users of the OpenLearn site. The OpenLearn site offers free 
access to Open University courses online. The next section presents some of the examples and 
describes the courses and the resources available in OpenLearn. The particular tasks given to users 
during the study are also outlined. 


3 The OpenLearn ‘Unit’ and the tasks for the feasibility study 

OpenLearn gives free access to some of the Open University course materials. The course materials 
are referred to as ‘learning units’. Each unit can be accessed through a web browser, and contains 
typical components such as the unit outline, the time expected to finish the unit, and ‘level’ of the unit 
(i.e. introductory, intermediate or advanced unit) (Figure 1). 
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Figure 1: An example of an OpenLearn learning unit: ‘ Global Warming’ 

Users can view the content of a unit in a sequence of ‘pages’ presented in the web browser window 
(Figure 2). Within the web browser, some of the content of a unit may include resources available as 
either embedded or as a text hyperlink that can be viewed i) as a PDF or Word document (e.g. Figure 
3) ii) a video or audio resource iii) as another link to a different resource. 
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Pages: <Previous 123456789 10 11 Next> 

Introduction 

This unit provides an introduction to global warming. We will be considering the history of global warming by looking at 
the pattern of ice ages and analyisis of recorded temperatures. We will aim to gather meaningful information from this 
data. We will briefly assess the impact and influence of humans on global warming and, finally, we will examine climate 
models and how to predict future changes. 


Learning Outcomes 

The learning outcomes for this unit are to: 

• Develop an understanding of the current evidence for 
global warming. 

• Model and apply the techniques of 'measuring' the Earth's 
temperature. 

• Understand the current warming in relation to climate 
changes throughout the Earth's history. 

• Explain factors forcing climate change, and the extent of 
anthropogenic influence. 

• Assess the 'best predictions' of current climate models. 


Pages: <Previous 123456789 10 11 Next> 

Figure 2: An example of the first page and the ‘page number navigational options’ of the 
learning unit ‘ Global Warming’ 
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3.2.2 Pollen diagrams and ancient climates 

The usefulness of fossil pollen to the scientist is that the types and proportions of 
pollen in a sample, such as Figure 3.10, can be compared with those produced by 
vegetation growing in present-day climates. The climate at the time the sample was 
formed can then be inferred. This brings us back to the main thread of this section, 
which is the pattern of temperature change on the Earth through time. 
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Figure 3.10 Pollen grains that have been carefully 
separated and extracted from a sample of clay from 
Marks Tey, near Colchester, Essex, seen magnified by 
a microscope. 


Figure 3: An example PDF document resource relevant to a learning unit (e.g. ‘ Global 
Warming’ ) 


In this research, a predefined concept of ‘browsing’ interaction takes place when users go though a 
page very quickly without thoroughly reading the content of a page whereas ‘learning’ interaction 
takes place when users read the content of the page. Three tasks were designed to investigate if it is 
possible to characterise the difference between these two types of interactions. The first task is to find 
out what users do when studying a course unit. A unit was chosen and used to investigate this. This 
learning unit was selected to be ‘generic’, of general interest, and to not require any background 
knowledge in order for users to go through the content. Thus, the unit chosen for this task is about the 
introduction to ‘Global Warming’ as this unit fits the criteria given to the users participating in the 
study. The feasibility study, which involved asking users to complete all three tasks, is designed for an 
hour and a half session as consented with the users. This unit chosen is designed to take 5 hours but 
only the first few pages of this unit are used in the study. The first task is: 
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‘Study’ task - Users were asked to go through the Global warming unit and to complete the 
first set of exercises. 

To compare what users do when they are ‘browsing’ the second task is: 

‘Browse’ task - Users were asked to browse through some of the contents of OpenLearn. 


And, as a combination of the two kinds of task, the third task is: 

‘Study-choice’ task - Users were asked to study a certain unit of their choice. 

The tasks were designed to investigate if it is possible to differentiate between ‘browsing’ and 
‘learning’ interactions. The tasks were refined and tested with two users in a pilot study. 


4 Eye-tracking and remote observation studies 

The aim of this research is to investigate ways to determine different kinds of users’ actions from 
matching observation based on eye-tracking and remote observations. It aims to identify differences 
between actions that can be described as browsing and as learning. Consequently, this research aims to 
provide some specifications for conducting indirect study of users’ interactions based on logs and to be 
able to identify browsing and learning actions based on these logged. Logged actions can be frequency 
of visit, and the time spent, in every page or resource. 

Two studies were designed to investigate the extent with which log actions can correspond to two 
types of actions. These two studies are eye-tracking and remote observation studies. The investigation 
focuses on whether observed actions from eye-tracking study and observed actions from the remote 
observation study be said to correspond to some patterns. 


The technologies and the observation techniques and the kinds of data corresponding to each of the 
studies are given in Table 1 below. 



Eye tracking study 
(Two participants) 

Remote study (Two participants) 

Technologies 

ClearView™ Eye-tracking system, Tobii™ 
Eye tracker, Digital video camera 

Webcams, Laptop, Netviewer™ (remote 
desktop sharing technology), Camtasia™ 
(screen capture technology) 

Technique/s 

Combined eye-tracking and think-aloud 
Collects what users say, do and see 

Think-aloud technique 
What users say and do 

Kinds of data 

Talk and video data 

Logs: Time spent visiting a resource, the 
resource visited, mouse actions, gaze data 

Talk and video data 

Logs: Time spent visiting a resource, the 
resource visited 


Table 1: The two studies undertaken to investigate the difference between browsing and learning 
logs 


Eye-tracking technique has been used by many to look at learner interactions (e.g. Gluck, 1999, San 
Diego & Aczel, 2007, Hansen, Hauland, & Andersen, 2001, Yoon and Narayanan, 2004). In this 
research, the eye-tracking study combined different techniques to capture what users say, do and see; 
and relates these with Togged data’ of users’ interaction. This set of techniques was established and 
tested based on a larger research by the first author (see San Diego, 2008). Figure 4 shows an example 
of two screens recorded in the eye-tracking study conducted. The researcher observes the actions in 
real-time in a laboratory. User actions and utterances (user video on the left of Figure 4); and screen 
activity with eye-tracking data (eye-tracking video on the right of Figure 4) were recorded and can be 
played in synchronisation. On the figure, the eye-tracking videos shows ‘saccades’ as lines - the path 
the eye took across the screen; ‘fixation’ as blobs - the place where the eye dwelled on the part of a 
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screen; and mouse clicks as ‘x’ marks - location of clicks. Examples of logged data are given in Table 
5 later in this paper. 



Fixations 


Mouse click location 

Saccades 







Figure 4: A screenshot of the videos recorded in the eye-tracking study 


Remote observation is still in its infancy as the technology remains limited to be able to capture real- 
time video of remote desktop via the internet and to send this real-time video signal back to the remote 
observer for recording ‘just-in-time ’ (Ivory & Hearst, 2001). Thus, the procedure and technologies for 
the remote observation study was tested and trialled several times with two colleagues. The kinds of 
data captured in this study have some similarities to the eye-tracking study. In the remote observation, 
user’s gazes cannot be recorded because the home computers lack the cameras that detect users’ eye- 
movements; and due to the limitation of the system where OpenLearn was developed (i.e. Moodle™ - 
an open-source course management system) logged mouse actions cannot be recorded. Using a 
remote-desktop sharing technology and screen capture software, it was possible to remotely capture 
what users’ say and do. Figure 5 shows a remote user video (left) recorded onto the researcher’s 
computer. The remote video and audio signal are transmitted via the internet and recorded in the 
researcher’s computer using Camtasia™. The researcher (right) watched the user’s interaction in ‘near- 
synchronous’ time during the observation session. The user is not in the same room as the researcher. 



Figure 5: A screenshot of the videos in the remote observation study 



Four users, who were aware of the OpenLearn site but do not have much experience with any of the 
units, participated in the study. Two users were assigned in the eye-tracking study and the other two in 
the remote observation study. Their activities were recorded as described above and their logs were 
captured. In the two studies, the instructions given in completing tasks were slightly varied due to the 
differences in technologies used and the procedures required for capturing data. 
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5 Differing actions when ‘browsing’ and ‘learning’ 

The videos from the two studies were watched several times before seeking to identify which kinds of 
actions can correspond to ‘browsing’ and ‘learning’. After having, watched the videos, several times, 
a pattern seems to emerge from relating the kinds of logged actions with users’ utterances and actions 
in both studies. The analysis of the videos suggests a difference between the interactions when users 
are asked to complete the browsing task and when users are asked to complete the study-choice task. 
Given the analysis between these two tasks, the following variations on users’ actions are derived in 
Table 2 below. 


Browsing task - Users were asked to browse 
through some of the contents of OpenLearn. 

Study-choice task - Users were asked to study a 
certain unit of their choice. 

In both eye-tracking and remote study , the users: 

Move mouse pointer fast almost the entire time of 
the task; 

screen display scrolls fast down and up 
repeatedly, then slows down in several occasions 
all throughout the task; 

Choose random pages with a unit and open 
resources at random; 

Open resource at random. 

In both eye-tracking and remote study , the users: 

Move mouse pointes fast at the first few seconds; 
(sometimes) drag mouse pointer from left to right 
along the text that is being read out loud; 
screen display scrolls down and up slowly with 
few occasions of scrolling up; 
choose a page following a certain sequence or 
visit related pages; 

Open resource as read out. 

Evidence from eye-tracking shows 

Eye-moves ‘ballistic’ (i.e. saccades moving left-to 
right, up-to-down) as text being read out loud are 
found in different sentences or different paragraph 
within a page of a certain unit. Eye-movements do 
not follow a specific pattern of fixation on 
contents within a webpage. 

Evidence from eye-tracking shows 

Indication of ‘reading’ may be represented by a 
‘worm-like’ saccadic movement from left to right 
(e.g. Figure 4). 


Table 2: Users’ actions in the ‘browsing’ and ‘study-choice’ tasks 


From Table 2, the ‘browsing’ action happens when the user, for example, is choosing content that is of 
interest to him or her. This is denoted by fast scrolling mouse-movement, random clicking, random 
word reading, random page flicking, random opening of resources, and coupled with ‘ballistic’ eye 
movement. The ‘learning’ action happens when the user, for example, has chosen a certain part to 
thoroughly read through. This action is denoted by a slow scrolling, slow mouse movement (may be 
from left to right along the line of a text), clicking of pages in sequence, opening of resources in 
sequence, and ‘worm-like’ movement of the saccades. These actions are further described in Table 
below. The ‘study-choice’ task both illustrated an indication of ‘browsing’ action described above 
when the users were choosing content then shifted to a ‘learning’ action when the user decided to go 
through a chosen content. These forms of interaction may possibly represent some of the typical 
actions that a user might engage with when they are ‘browsing’ and ‘learning’ a unit in the OpenLeam. 

An example extract is given below of a user’s talk while ‘browsing’. 

The user looked at the range of topics (00:00:46:15) “Oh I'm torn between going for something that 
interest me and going for something that I think might be of use to me... I’ll combine the two and go 
for 'Science and Nature'” 

The user then scrolled up and down the list of sub topics and pausing when (00:01 :01 : 1 8) “Nutrition. 
Vitamins and Minerals', 'Evolution through Natural Selection'... that looks interesting...” 

The user looked at another sub-topic then decided to look at the first choice (00:01:30:00) “Ok... I'm 
tempted by both of them but I know more about earthquakes than I do about evolution... So I'm going 
to look at evolution first.” 
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After deriving some descriptors for actions associated with browsing and learning, the logs were 
scrutinised and analysed. The following visualisation of models that can be interpreted as browsing 
and learning are given in the next section. 

6 Specifying kinds and models of browsing and learning actions 
based on logs 

The OpenLearn Units are created in a platform that can automatically generate logged actions such as 
time when a user visited a resource, the kind of action (i.e. viewing a content of a certain Unit, etc), 
and also the label given to a resource. Table 3 is an example which shows a log of a user’s interaction 
in the browsing task of a certain Unit ‘Teaching for good behaviour’. The description on the last 
column (right) has been added to indicate the Page number and the resources available within each 
Page. 


Time 

Action 

Resource label 

Description 

12:56:00 

course view 

Teaching for good behaviour 

Unit Page (Page 0) 

12:57:00 

resource view 

5. Developing & engaging lessons 

Page 5 

12:57:00 

resource view 

References 

Page 7 

12:57:00 

resource view 

4. Lesson content 

Page 5 (has web link 
resource and an activity) 

12:57:00 

resource view 

3. Lesson delivery 

Page 4 (has an animation 
embedded, a PDF 
document, and an 
activity) 

12:57:00 

resource view 

2. Lesson format 

Page 3 (has two web link 
resources and an activity) 

12:57:00 

resource view 

1. Teaching and behaviour 

Page 2 

12:57:00 

resource view 

Introduction 

Page 1 

12:58:00 

resource view 

5. Developing & engaging lessons 

Page 6 (has an activity 
and a PDF document) 

12:58:00 

resource view 

1. Teaching and behaviour 

Page 2 

12:58:00 

resource view 

Introduction 

Page 1 

13:02:00 

course view 

Teaching for good behaviour 

Unit Page 

13:02:00 

resource view 

3. Lesson delivery 

Page 4 

13:02:00 

course view 

Teaching for good behaviour 

Unit Page 


Table 3: The expected sequence of the learning Unit 


By examining logged actions in Table 3, it is easy to extract the frequency of visits in every page. 
However, the platform does not include in the log the resources that users open. Also, by careful 
inspection of the logged time, it is also difficult to accurately determine the amount of time spent in 
each page as the time is not recorded in sufficient detail. 

By chronologically examining the sequence of pages visited, the log shows that the user did not visit 
the pages in the designed sequence. In the case shown in the table it is possible to identify a browsing 
interaction from the logged actions. For example, the user interactions at 12:57 can be denoted as 
browsing because seven different pages were visited within one minute. Each of the pages consisted of 
texts and resources that cannot be read within one minute. On the other hand, log of time and pages 
visited between 12:58 and 13:02 are indicative of learning action Page 1 because three minutes is a 
reasonable time to read the content of that page. However, it may be argued that the user could have 
their attention elsewhere. The eye-tracking study records other kinds of logs that can reduce this 
limitation. For example, mouse movement logs and slow scrolling from up to down can indicate that 
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the user is present in front of the computer; and that the logged action may indicate that a user is going 
through the content of a page. 

The rest of this section shows some possible visualisations of how logs can be transformed into a 
representation that can help characterise whether ‘browsing’ and ‘learning’ action are happening. It 
also illustrates the kinds of browsing and learning actions taking place when users are studying by 
analysing the four users’ logged actions in completing the ‘study task’ (discussed later in this section). 
First, the sequence of page visits based on logs is illustrated as to how this can also be used to examine 
different sets of actions. 

To illustrate the value of a sequence visualisation based on logs, an example logged action from one of 
the users completing the study task is used below. The study task is to study the Global Warming Unit 
and complete the ‘first exercise’. Like other learning units in OpenLearn, the Global Warming Unit is 
likely to have been designed with the expectation that users would go through the unit following a 
certain sequence. First, users are expected to read Page 1, the introduction and the learning outcomes 
of the unit; then to go through Page 2; and so on. Table 4 shows the sequence that users are expected to 
follow. The first (left) column of Table 4 shows the expected time users may spend for each of the 
resources from Page 1 up to the ‘first exercise’ of the Unit; threshold time, in minutes, is the expected 
amount time that users may spend going through the content within a page. The middle column gives 
the resource and the third column shows the corresponding URLs. It is expected that users may take 
from 15 minutes to 25 minutes to go through this part of the unit. 


Threshold 
(in minutes) 

Resource 

URL 

1.5 to 2.5 

Page 1 : Introduction and 

http://openleam.open.ac.uk/mod/resource/view.php?id 

minutes 

learning outcomes 

=171493 

2 to 3 

Page 2: Natural climate change 

http://openleam.open.ac.uk/mod/resource/view.php?id 

minutes 

=171495 

.5 to 1 

First paragraph of Page 3: A 4.6 

http://openleam.open.ac.uk/mod/resource/view.php?id 

minute 

billion-year history 

=171497 

1 to 2 

View PDF: chart of the 

http://openleam.open.ac.uk/file.php/1526/sci ski 07i. 

minutes 

temperature of the earth 

pdf 

.5 to 1 

Second paragraph of Page 3: A 

http://openleam.open.ac.uk/mod/resource/view.php?id 

minute 

4.6 billion-year history 

=171497 

5 to 7 

View PDF: Glaciers past and 

http://openleam.open.ac.uk/file.php/1526/sci ski 05t. 

minutes 

present (five Pages) 

pdf 

5 to 7 

View PDF: Pollen diagrams and 

http://openleam.open.ac.uk/file.php/1526/sci ski 06t. 

minutes 

ancient climates (five Pages) 

pdf 

.5 to 1 

Read exercise of Page 3: Past 

http://openleam.open.ac.uk/mod/resource/view.php?id 

minute 

temperatures 

=171497 

Table 4: T1 

le expected sequence of the learning Unit ‘Global Warming’ 


The sequence above can be represented in a flow diagram as in Figure 6. This kind of visualisation 
helps to make clear the kinds of resources and the amount of text, picture, diagram, video, document, 
etc. that a unit consists. The number and amount of visits can also be embedded on each page (see for 
example in Figure 6 below, Page 2 and Page 3, where n represents the frequency of visit, and t 
represents the amount of time spent). The accumulated number of visits across a set of user in every 
page may indicate the least popular Page and most popular one visited by users. An alternative 
visualisation may help and is presented in the next figure. 
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n = 2; t = 1 min 


Pages: <Previous 123456789 10 11 Next) 

2. A 4.6 billion- year history 

Climate change is a natural process of warming and cooling that has occurred all through the Earth's history. 
Throughout geological time there have been iiot- house' penods and ice ages. In order to understand the current 
situation, it is necessary to have some sense of context and perspective, from historical and geological time- scales. 
The document below shows a chart showing a generalised temperature history of the Earth. 


Click on View document' to see the chart 

View document 

We are currently enjoying an interglacial period of an ice age that began approximately two mifton years ago. (Ice 
ages are composed of colder 'glacial' and warmer 'interglacial' penods.) This is at least the seventh ice age m the 
Earth's 4.6 billion- year history. 

Ounng the Cretaceous (65-147 Ma) the whole Earth was up to ♦IS (C warmer than at present, with tropical forests 
covenng Antarctica, whereas dunng the Quaternary (12 Ma) ice sheets spread across much of Europe and the mean 
surface temperature was up to (S (C colder than at present. The range of these changes is much greater than the 
observed increase in temperature over the past century (+0.6 ±0.2 (C) and predictions for the next hundred years (+3 
±1.5 (C). 

Click on View document' to read ‘Glaciers past and present' 

Vi^w donumpnt 

Click on View document' to read Pollen diagrams and ancient climates' 

View document 

Activity 1 Past temperatures: before measurement, before people 
Measuring temperature is a relatively recent affa*. Galileo GaUei invented 

Ins thermometer in 1597, but the 



Figure 6: The sequence flow diagram of the learning Unit 


The frequency of visit and the amount of time spent in each resource can be represented as a ‘heatmap’ 
as shown in Figure 7 where a colour coded overlay appears over a page; the lighter the colour the less 
attention and the time spent, the darker the colour the greater the attention and time. This kind of 
visualisation in possible in the eye-tracking system (where the heat map in the eye-tracking system 
represents the amount of time users dwell on the part of the screen). There are now other technologies 
transforming logged visit and time of visit and even mouse action and movements as heatmaps (see 
e.g. http://www.clickdensity.com/ and http://diligint.com/) 



Figure 7: An example of a heat map visualisation 


It may also valuable to visualise the sequence followed by a user to help examine the degree of match, 
or mismatch, between a perceived pedagogic sequence and the actual sequence followed by a user. A 
possibly better visualisation than Figure 6 is the use of timeline visualisation. An example of a timeline 
is given in Figure 8. Figures 8 show ‘bars’ in different colours where each colour represents a page, the 
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length of the bar represents the proportion of time spent. This representation is better than Figure 6 
because it can show both the sequence and the amount of time spent in a more visual way; however, it 
does not pictorially represent the kinds of resources. Figure 8 is the timeline generated from 
transforming Table 4 above which represents the pedagogic sequence design of a course developer. 



- Unit page 



- Page 1 (Introduction) 


-PDF 


- Page 2 (Climate change) 


-PDF 

7 ^ 

■ ■■ a 1 
■■■■■a' 

■ 

1 - Page 3 (4.6 billion-year history) 


-PDF 


(Chart; earth’s temperature) 
(Glaciers past and present) 
(Pollen diagrams and ancient) 


In the legend: the width of the bar represents a minute 
Figure 8: The Timeline representing a visual sequence flow diagram of the learning Unit 


An extract of an actual user log based on the eye-tracking system is given in Table 5 below. Table 5 
log differs from Table 3 in terms of: i) time is in milliseconds (i.e. 100 milliseconds = 1 second) ii) 
start and end time of visit is given (i.e. URLVisibleBegin means start of the visit and URLVisibleEnd 
means end of visit) iii) mouse clicks with a Page is given (e.g. LMouseButton means a left click). This 
kind of log capture is not unique to the eye-tracking systems. There are also others collecting similar 
kind of information (e.g. http://diligint.com/ and http://www.mangold.de/LogSquare-V3-0. 28. 0.html ) 


Time 

Event 

Description 

11437 

URLVisibleBegin 

http://openleam.open.ac.uk/course/view.php?id=1526 

14418 

LMouseButton 


15027 

URLVisibleEnd 

http://openleam.open.ac.uk/course/view.php?id=1526 

15028 

URLVisibleBegin 

http://openleam.open.ac.uk/mod/resource/view.php7kU171493 

17456 

LMouseButton 


21164 

LMouseButton 


22609 

LMouseButton 


23216 

LMouseButton 


23437 

LMouseButton 


24574 

LMouseButton 


25301 

URLVisibleEnd 

http://openleam.open.ac.uk/mod/resource/view.php7kU171493 

25304 

URLVisibleBegin 

http://openleam.open.ac.uk/mod/resource/view.php7kU171495 

26925 

LMouseButton 


28497 

LMouseButton 


28733 

URLVisibleEnd 

http://openleam.open.ac.uk/mod/resource/view.php7kU171495 

28735 

URLVisibleBegin 

http://openleam.open.ac.uk/mod/resource/view.php7kU171497 


Table 5: The Timeline representing a visual sequence flow diagram of the learning Unit 


By aggregating the sequence followed by a number of users, it may be possible to inform the course 
designer on how the sequence should be redesigned. Figure 9 shows the actual sequence followed by 
the four users in completing the study task. In Figure 9, User 2 can be discounted as this user withdrew 
from doing the study task but agreed to browse though the content. User 2 withdrew when asked to do 
an exercise. As the figure shows, User 2 spent most of the time on the ‘Unit Page’ but also browsed 
through and looked at the exercise in Page 3. 

By inspecting the timelines in Figure 9, Page 3 appeared several times as the expected sequence 
suggest. However, by carefully inspecting the proportion of the bar, users did not spend much time in 
other pages before the first visit of Page 3. This may suggest that User 1, 3 and 4, browsed through 
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first few pages because the length of the bar is less than a minute. This length of time is not enough to 
go through the content of each of the page. 


Expected design sequence 




: = : 

: = : 

: = : 

: = : 

: = : 

: = ~ 
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■ 











■■.y n.n.MMMM.nn * * * * * • # * * i 



User 1 
User 2 
User 3 
User 4 






r:« * * t:) 





■■■■■■■ 


- Unit Page 

- Page 1 (Introduction) 

- Page 2 (Climate change) 

- Page 3 (4.6 billion-year history) 



(Chart; earth’s temperature) 
(Glaciers past and present) 
(Pollen diagrams and ancient) 


In the legend: the width of the bar represents a minute 


Figure 9: The sequence followed by the four users of in the study task 


Triangulating logs with other forms of research data can help interpret and validate the kinds of actions 
found. For example, in Figure 9 above, the three users were found browsing the Pages because they 
were looking for the exercise. The talk data and the scrolling into the area of where the exercise can be 
found on the part of the screen validate this. 

User 4 visited the Unit Page quickly click the ‘Introduction link’ which brings him to Page 1 . he 
read out random words on Page 1 , then clicked on page 2, then Page 3, and then he said, 

(00:00:46: 15) “I would like to see the exercise. . .Err. . . Ok. . . I know what I am going to look 
for.” 


Users 1 and 3 have performed similar actions. The three users looked for the page where the exercise 
can be found. After having found the exercise, the length of the bar changed to a longer bar. This 
indicates more time spent on each the pages. The length of time commensurate to the amount of time 
users may spend looking at the pages. However, it can be argued that by just relying on this 
visualisation, it may not precisely represent that Teaming’ is taking place. A way to reduce this is to it 
with other forms of data such as logs of mouse clicks, mouse movement, scrolling of the display. 
Additionally, other researcher techniques, e.g. survey questionnaires and online interview questions, 
may further help interpret the data based on logs. 


7 Concluding thoughts and recommendations for future work 

The purpose of this research is to investigate the possibility of classifying different online interactions 
actions from an eye-tracking study and a remote observation study. This research offers some insights 
for describing the difference between browsing and learning actions by examining visualisations of 
recorded user activity. 

Although, digital technologies may offer possibilities to research learner-interactions there are many 
challenges that need to be addressed in future research. Among others, for example, ethical 
considerations for remote observation studies can be more complicated than the ethics concerning 
traditional observation. Tmst and rapport with participants may not be easy to establish due to the 
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‘physical’ absence of the researcher. There are also methodological challenges such as the 
interpretation of logs without the presence of user video capture, in such cases recorded website 
interaction times may not really represent an actual time as users may not be in front of their screen 
between the times of different actions. 

While others have distinguished a ‘learning’ action as frequency of visit and time of visit to online 
resources (e.g. Sheard et al., 2003), this research has extended this by specifying the possibility of 
determining differences between different kinds of online interactions. The data from four participants 
of the two small studies (i.e. an eye-tracking study and a ' remote observation ’ study) seem to offer 
evidence for identifying the difference between users flicking through the web pages in “browsing” 
action, and when they are more engaged in “learning” action. 

This research has provided some examples of visualisations that could show a representation of the 
perceived pedagogic sequence of learner interaction with a course material as thought of by course 
designers and the visual representation of actual user interaction based on logs. By analysing user 
interactions based on visualisations of logs triangulated with users’ utterances, the evidence suggests 
that although an OpenLearn unit may have been designed to follow a certain pedagogic sequence, logs 
show users may not follow the same sequence. For example, the two users in this research ‘jumped’ to 
the webpage where the assessment questions are. It seemed that the users performed some sort of 
‘answer searching’ strategy. Without constraining users of specific instructions in performing 
interactions, mismatch between these visualisations might provide a way to adapt the design of the 
sequence of online learning materials. 

This research has attempted to illustrate the feasibility of examining, identifying and observing 
‘learning’ and ‘browsing actions based on user logs. While it is possible classify different kinds of user 
actions, this research has also considered ways to transform logs from different systems into a 
visualisation that may be more easily interpretable than the ‘raw’ text-form of user logs. One of the 
limitations of determining user interactions from logs is whether users are actually looking at the 
website based on logs. This can be reduced if there is a technology that captures sufficient detail in the 
logs (frequency of webpage visits, start and end time of webpage visits in milliseconds, duration of 
visits, mouse pointer movement, mouse and keyboard actions, and if possible screen capture of remote 
user’s computer) that can contribute to measuring when user action is happening associated with the 
user being present in front of the computer screen. While this does not rule out the possibility that 
users are not reading the content, the logs can be coupled with inputs of estimated time users would 
take to go through a webpage; and whenever the user log goes extremely beyond the estimated time 
without any mouse movement (e.g. slow scrolling up to down, clicking, movement left to right), this is 
an indicator that users are not active and data may not be valid. Even so, the kinds of interactions based 
on logs have to be carefully interpreted and complemented and triangulated with other forms of 
research data, such as that from surveys, interviews, questionnaires, and remote screen capture. 

Capture of logs will also be improved if webpages can be divided into several sections (i.e. slicing a 
page of a unit into several parts depending on the kind of resource present), and recording of logs by 
different sections. Such detailed log records would also help identify interactions with different kinds 
of resources and the elements within a webpage. Interpreting user actions could not only improve our 
understanding as researchers, but also lead to benefits for users. In the future, systems may be able to 
change the sequence of pages depending on the pattern of user interactions of a majority of users and 
feedback this information to designers to help them improve online learning resources. 
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